Noname manuscript No. (will be inserted by the editor) Abstract Selecting an optimal set of icons is a crucial step in the pipeline of visual design to structure and navigate through content. However, designing the icons sets is usually a difficult task for which expert knowledge isrequired. In this work, to ease the process of icon set selection to the users, we propose a similarity metric which captures the properties of style and visual identity. We train a Siamese Neural Network with an online dataset of icons organized in visually coherent collections that are used to adaptively sample training data and optimize the training process. As the dataset contains noise, we further collect human-rated information onthe perception of icon's similarity which will be used for evaluating and testing the proposed model. We present several results and applications based on searches, kernel visualizations and optimized set proposals thatcan be helpful for designers and non-expert users while exploring large collections of icons. Keywords Iconography · Illustration · Visualization · Appearance Similarity · Machine Learning 1 Introduction Visual communication is one of the most important ways to share and transmit information [34,33]. In the same way as words are used for verbal communication, symbols or icons are the elements used to convey information ina universal and ubiquitous language [1, 20]. Icons are key elements to structure visual content and make it more appealing and comprehensible. Style and visual identity are preserved for each collection.